I just finished the first version of partitioned namespace and it is ready for review now. When I implemented it, I found the original design need some update, here is the reason.

Explanation of the New Design

Compared to the previous partitioning spec, the current implementation introduces two key changes. Below I’ll explain the motivation behind these adjustments and would love to get further feedback and discussion.

1. Introducing TableSpec

In the earlier partition spec design, all tables shared the same schema, which was stored in the table metadata. A PartitionedNamespace contained multiple sub-namespaces named v{i}, representing different versions of the partition spec.

The problem with that design is that it makes schema evolution difficult. We cannot atomically modify the schema of all tables without breaking the semantic guarantee that “all tables share the same schema.”

Even if we support multi-partition transactions in the future, we would still be blocked at the step of updating the schema stored in table metadata, because updating table metadata itself does not have transactional semantics.

To solve this problem, I propose introducing a new abstraction: TableSpec, which encapsulates both the schema and the partition spec.

It is defined as:

pub struct TableSpec {
    id: i32,
    schema: ArrowSchema,
    partition_spec: PartitionSpec,
}

TableSpec is stored as a first-level sub-namespace under the PartitionedNamespace root. The v{i} naming convention corresponds to the id, while the schema and partition_spec are stored in the namespace properties.

Whenever a schema evolution or partition spec evolution occurs, we create a new sub-namespace. This operation is purely metadata-level. In manifest namespace, creating a namespace is atomic.

New data must follow the new schema (i.e., tables are created and written under the new TableSpec namespace).

Existing data does not need to be modified and continues to use the previous schema and partition spec.

Another advantage of this design is that it makes it relatively straightforward to support branching functionality in the future.

The overall structure can be visualized as follows:

2. Using Deterministic Names for Namespaces

In the previous partition spec design, the name of a partition namespace was a 16-character base36 string. Any type of partition value would be mapped to such a 16-character base36 string.

The issue with this design is that it makes it difficult to resolve concurrency conflicts. In distributed scenarios, we may need to concurrently create tables and namespaces with the same partition values. Since partition values effectively serve as business primary keys, this becomes a consistency challenge.

Originally, I considered leveraging Lance’s merge-insert deduplication capability to enforce uniqueness at the business key level. However, merge-insert in Lance requires that:

• A column be explicitly defined as a primary key
• The primary key column must be non-null

In our current design, the __manifest partition field column must be nullable. Additionally, partition fields can evolve over time, meaning we cannot reliably define a fixed initial primary key column. This makes it infeasible to use merge-insert deduplication directly on partition fields.

To address this, I propose using deterministic names for partition namespaces. At each level, the namespace name is derived from the serialized string representation of the partition value.

With this approach:

• Namespace identity becomes deterministic and directly tied to partition values.
• We only need to define a primary key on the object_id column.
• We can then leverage merge-insert to resolve concurrency conflicts when creating new namespaces and tables.

This simplifies concurrency handling while preserving correctness in distributed environments.

Codecov Report

❌ Patch coverage is 66.99029% with 136 lines in your changes missing coverage. Please review.

📢 Thoughts on this report? Let us know!